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Optimisation and Fixed Point 


Let f: H — R be differentiable. 


Consider the problem: Find xX such that 


f(%) = min f(x) 
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Optimisation and Fixed Point 


Let f: H — R be differentiable. 
Consider the problem: Find X such that 

f(X) = min F(x) 
Equivalent problem: Find X such that 


VF(X) =0 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations OBI 1, June 2023 2/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
@0000000 00000 


Optimisation and Fixed Point 


Let f: 2 > R be differentiable. 
Consider the problem: Find X such that 
f(X) = min F(x) 
Equivalent problem: Find X such that 
Vf(x) =0 
Other equivalent problem: Find % such that 
TX=xX, where T=/-—Vf 
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Optimisation and Fixed Point 
Let f,g: H > R. Suppose that f is differentiable. 
Consider the problem: Find xX such that 


(f + g)(X) = min f(x) + g(x) 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations OBI 1, June 2023 3/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
O@000000 fotete} foto} 


Optimisation and Fixed Point 


Let f,g: H > R. Suppose that f is differentiable. 


Consider the problem: Find xX such that 


(f + g)(X) = min f(x) + g(x) 


Equivalent problem: Find X such that 


0 € O(f +. g)(X) = VF(X) + Og(X) 
=> X=(1— dg) (14+ VF)(%) 
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Optimisation and Fixed Point 


Let f,g: H > R. Suppose that f is differentiable. 
Consider the problem: Find xX such that 
(f + g)(X) = min f(x) +. g(x) 
Equivalent problem: Find X such that 
0 € O(f + g)(X) = VF(X) + Og(X) 
=> X= (1 — dg) (14+ VF)(%) 


Other equivalent problem: Find % such that 


TX =, where T = (/ — dg) +(1+ VF) 
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Fixed Point Iterative Schemes 


For T: H SH, find X € H such that TX = X. 
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Fixed Point Iterative Schemes 


For T: H > H, find X € H such that TX = X. 


Picard Iterations: 


Xk+1 = TX, 
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Fixed Point Iterative Schemes 


Oh Wicott— tt indie = et cUchn that. x — x. 


Picard Iterations: 


Xk41 = Tx 
Krasnoselskii-Mann Iterations: 


Xeei = (1 — Ag) ee FAR De 
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Fixed Point Iterative Schemes 


Oh dict tt indie = et cUchn that. Tx — x. 


Picard Iterations: 


Xk41 = Tx 
Krasnoselskii-Mann Iterations: 


Xk-1 = (1 — Ag) x + AK TxK 


Advantages of KM iterations over Picard iterations: 
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Fixed Point Iterative Schemes 


Oh dict — tt indie = et cUchn that. Tx — x. 


Picard Iterations: 


Xk41 = Tx 
Krasnoselskii-Mann Iterations: 


Xk-1 = (1 — Ag) x + AK TxK 


Advantages of KM iterations over Picard iterations: 


e Allows for a broader class of operators. 
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Fixed Point Iterative Schemes 


Oh dict — tt indie = et cUchn that. Tx — x. 


Picard Iterations: 


Xk41 = Tx 
Krasnoselskii-Mann Iterations: 


Xk-1 = (1 — Ag) x + AK TxK 


Advantages of KM iterations over Picard iterations: 
e Allows for a broader class of operators. 
@ Includes more diverse algorithms. 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xk41 = (1 _ Ak) Xk + A, TX, 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkar = (1 = An)xK + AK TX 


Heavy-Ball Acceleration: 


Vk = Xk + aK(xXk — Xk-1) 
Xegq = (L-Ag) Ve FAK TX 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkar = (1 = Ag) xR + AK TX 


Heavy-Ball Acceleration: 


Vk = = Xk + Ok ( XK — XK-1) 

Xktn = (L=Ak) VK + AR TX 
Nesterov Acceleration: 

Ze = Xk + Bu (Xk — Xk-1) 

Mea = (l=Aglze + Acie 
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Acceleration Methods 
Krasnoselskii-Mann Iterations: 


Xkp1 = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xk4t = (1L—Ak) VK +AK TZ 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations 


OBI 1, June 2023 5/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
O00@0000 00000 


Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkar = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xe = (1L—Ak) VK + AKT ZK 


Special cases: 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkp1 = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xk4t = (1L—Ak) VK + AKT ZK 


Special cases: 
e@ When a, = 6, = 0, we get no acceleration 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkp1 = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xk4t = (1L—Ak) VK + AKT ZK 


Special cases: 
e@ When a, = 6, = 0, we get no acceleration 
@ When ax = Gx, we get Nesterov acceleration 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkp1 = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xk4t = (1L—Ak) VK + AKT ZK 


Special cases: 
@ When a, = 6, = 0, we get no acceleration 
@ When a, = (x, we get Nesterov acceleration 
@ When (; = 0, we get Heavy-Ball acceleration 
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Acceleration Methods 


Krasnoselskii-Mann Iterations: 


Xkar = (1 = An) XK + AK TX 


General Inertial Scheme: 


Vk = XK + K(X — Xk-1) 
Zk = Xe oeOG a xe) 
Xe = (1L—Ak) VK + AKT ZK 


Special cases: 
@ When a, = 6, = 0, we get no acceleration 
@ When a, = (x, we get Nesterov acceleration 
e@ When £; = 0, we get Heavy-Ball acceleration 
@ When a, = 0, we get reflected acceleration 
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Adding Perturbations 
Krasnoselskii-Mann Iterations: 


Xk41 = (i _ Ak) Xk + A, TX, 
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Adding Perturbations 
Krasnoselskii-Mann Iterations: 

Xka1 = (1 = An)xK + AK TX 
Perturbed Krasnoselskii-Mann Iterations: 


Xk41 = (1 _ Ak) Xk +A, TX, + EK 
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Adding Perturbations 
Krasnoselskii-Mann Iterations: 
Xka1 = (1 = An)xK + AK TX 
Perturbed Krasnoselskii-Mann Iterations: 
Xka1 = (1 — Axx + An Tx + Ek 


Let T, = T + Ex where E, — 0. 
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Adding Perturbations 
Krasnoselskii-Mann Iterations: 
Xka1 = (1 = Ak) xK + AK TX 
Perturbed Krasnoselskii-Mann Iterations: 
Xko1 = (1 — Axx + An Txk + Ek 


Let T, = T + Ex where E, — 0. KM iterations with the operators T, 
replacing T may be written as 


Xk = (1 — An) AAR TX 
(1 — Ag) Xk + AK TX A AK ERX 
(1 = Ak) Xk + Xk TXk + Ek, 


with e, = Ac ERX. 
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Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Vk = Xk + aK (Xk — Xk-1) + Ex 
Zk = Sp Bele = Xen) 4 Pe 
Xegt = (1L—Ak) ye AK TZ + Ok 
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Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Yk = XK + an(xXk — Xk-1) + Ex 
Zk = Ke Pile — Mea) pe 
Xk4t = (1L—Ak)¥K HAR TZ + OK 


Theorem (Weak Convergence) 


Let T: H — H be nonexpansive such that F := Fix(7) 4. Under mild 
conditions, (xx), (YK) and (zx) converge weakly to a same point in F. 
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Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Yk = XK + K(X — Xk-1) + Ex 
Zk = Ke Pile — Mea) pe 
Xk4t = (1L—Ak)¥K HAR TZ + OK 


Theorem (Weak Convergence) 


Let T: H — H be nonexpansive such that F := Fix(7) 4 @. Under mild 
conditions, (xx), (y«) and (zx) converge weakly to a same point in F. 


Theorem (Strong Convergence) 


Let T: H — H be contractive such that Fix( 7) = {p*}. Under mild 
conditions, (x) converges strongly to p*. 
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Extensions 


The previous theorems may be extended by 
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Extensions 


The previous theorems may be extended by 


@ replacing “nonexpansive” and “contractive”’ by “quasi-nonexpansive” 
(K =1) and “quasi-contractive” (K < 1): 


|| Tx — pl < K]|x— pl], Vx € H, p € Fix(T) 
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Extensions 


The previous theorems may be extended by 


@ replacing “nonexpansive” and “contractive”’ by “quasi-nonexpansive” 
(K =1) and “quasi-contractive” (K < 1): 


|| ™ — pl| < K||x— pl], Vx € H, p € Fix(T) 
@ taking a family of operators T,: H — H such that 


F := (\ys1 Fix( Tk) #0 (Weak case) or Fix(T,) = {p*} for all k > 1 
(Strong case). 
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Extensions 


The previous theorems may be extended by 


@ replacing “nonexpansive” and “contractive”’ by “quasi-nonexpansive” 
(K =1) and “quasi-contractive” (K < 1): 


|| Tx — pl < K||x— pl], Vx € H, p € Fix(T) 


@ taking a family of operators T,: H — H such that 
F :=(\ys1 Fix( Tk) #0 (Weak case) or Fix(T,) = {p*} for all k > 1 
(Strong case). 


@ taking a family of operators T,: H — H such that 
F := Li(Fix(7,)) 4 0 (Weak case) or Fix(7T,) = {px} such that 
Pk — p* (Strong case). 
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Extended Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Vk = Xk + an(x — Xk—1) + Ex 
Zk =  Xq + Bele — Xe—1) + Oe 
Mea = (L=An)ye tp Actizee +O 
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Extended Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Vk = XK + an(xk — Xk-1) + Ex 
Zk = Ket Bie — me-1) + Pe 
Xk4d = (1—Ak)ye KAR TZ + Ok 


Extended Theorem (Weak Convergence) 


Let T,: H — H be quasi-nonexpansive such that F := Li(Fix(7,)) 40. 
Under mild conditions, (xx), (yx) and (zx) converge weakly to a same 
point in F. 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations OBI 1, June 2023 9/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
O000000®e O0000 [ele) 


Extended Convergence Theorems 


Perturbed General Inertial KM Iterations: 


Vk = XK + an(xk — Xk-1) + Ex 
Zk = Ket Bie — e-1) + Pe 
Xk4d = (1—Ak)yK KAR TZ + Ok 


Extended Theorem (Weak Convergence) 


Let T,: H — H be quasi-nonexpansive such that F := Li(Fix(7,)) 4 0. 
Under mild conditions, (xx), (yx) and (zx) converge weakly to a same 
point in F. 


Extended Theorem (Strong Convergence) 


Let T,: H — H be quasi-contractive such that Fix(7,) = {px} and 
Pk — p*. Under mild conditions, (xx) converges strongly to p*. 
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Application to Optimisation 


Problem 
Let f,g:H > RU {+oo}, hi: HOR, and L: H > H. Find 


min POQISE ACH SE Iai) 
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Application to Optimisation 


Problem 
Let f,g:H > RU {+oo}, hi: HOR, and L: H > H. Find 


min POQISE ACUI SE ate) 


This may be solved by the three-operator splitting method (Davis, Yin, 
2017): 


Tk = 1 — proxy, + proxy, ¢ 0 (2prox,,. — | — pxl* o Vho Lo prox,, ,). 
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Application to Optimisation 


Problem 
Let f,g:H > RU {+oo}, h: HOR, and L: H > H. Find 


min POQISE AC HISE ates) 


This may be solved by the three-operator splitting method (Davis, Yin, 
2017): 


Yk = Xk + OK (XK — XK-1) + Ek 

Ze = Xk + BR (Xk — Xk-1) + Pk 

a = prOXy,¢(Zk) 

Z = prox, ¢(2z¢ — Z% — pxl* o Vho L(zé)) 
Xktl = Vk +An(zi — 2) + % 
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Image Inpainting? 


Original Image 
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Image Inpainting? 


Original Image Corrupt Image 
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Image Inpainting? 


Original Image Corrupt Image Recovered Image 
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Image Inpainting? 


Original Image Corrupt Image Recovered Image 


Figure: Not obtained through described algorithm! 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations OBI 1, June 20: 11/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
00000000 fet Telete} 


Image Inpainting? 


Original Image Corrupt Image Recovered Image 


Figure: Not obtained through described algorithm! 


Mathematical Formulation 


1 
I x Th hss : Z EF Z He 
septa | gIAZ ~ Zewel? + ol Zaylle + olZayl} 
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Visual Results 


Original Image 


Corrupt Image 


Heavy-Ball 


Nesterov 


Figure: Process obtained with p = 1.8, A= 1.3, and o =0.5. 
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Convergence Plots 


Xe =Xe-1/? ITeZie = Yel? 
1034 10° 3 
10? 4 10? 3 
1014 10:1 
10° a T T T T T 10° a T T T T T 
1?) 10 20 30 40 1) 10 20 30 40 
Iteration (k) Iteration (k) 
[Xk-X"? IF(Xx) =X" |? 
5 102° 4d 
10° 4 
10° 3 
44 
10 10° 4 
1) 10 20 30 40 
Iteration (k) Iteration (k) 
— Heavy-Ball --- Perturbed Heavy-Ball —— Nesterov --- Perturbed Nesterov 


Figure: Process obtained with p = 1.8, A= 1.3, ando = 0.5. 
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Result Based on Algorithm 


Original Image Corrupt Image Recovered Image 


Figure: Obtained through perturbed inertial algorithm. 
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Conclusion 


Summary: 
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Conclusion 
Summary: 


e@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 
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Conclusion 
Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


e@ They incorporate multiple types of acceleration. 
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Conclusion 
Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


e@ They incorporate multiple types of acceleration. 


@ They account for (rounding) errors and/or inexact computations. 
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Conclusion 


Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


e@ They incorporate multiple types of acceleration. 
@ They account for (rounding) errors and/or inexact computations. 


@ They also allow for operators not sharing a common fixed point. 
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Conclusion 


Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


They incorporate multiple types of acceleration. 
They account for (rounding) errors and/or inexact computations. 


They also allow for operators not sharing a common fixed point. 


They converge weakly and strongly under mild conditions. 
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Conclusion 


Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


They incorporate multiple types of acceleration. 
They account for (rounding) errors and/or inexact computations. 
They also allow for operators not sharing a common fixed point. 


They converge weakly and strongly under mild conditions. 


They include the three-operator splitting method, used for 
optimisation problems such as the image inpainting problem. 
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Conclusion 


Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


They incorporate multiple types of acceleration. 
They account for (rounding) errors and/or inexact computations. 
They also allow for operators not sharing a common fixed point. 


They converge weakly and strongly under mild conditions. 


They include the three-operator splitting method, used for 
optimisation problems such as the image inpainting problem. 


Further possible research: 
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Conclusion 


Summary: 


e@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


They incorporate multiple types of acceleration. 
They account for (rounding) errors and/or inexact computations. 
They also allow for operators not sharing a common fixed point. 


They converge weakly and strongly under mild conditions. 


They include the three-operator splitting method, used for 
optimisation problems such as the image inpainting problem. 


Further possible research: 


e Study of the rates of convergence. 


Daniel Cortild, University of Groningen Perturbed Inertial KM Iterations OBI 1, June 2023 15/16 


Perturbed Inertial KM Iterations Optimisation Conclusion 
00000000 00000 @0o 


Conclusion 


Summary: 


@ Perturbed Inertial KM Iterations generalise previously known 
algorithms. 


They incorporate multiple types of acceleration. 
They account for (rounding) errors and/or inexact computations. 
They also allow for operators not sharing a common fixed point. 


They converge weakly and strongly under mild conditions. 


They include the three-operator splitting method, used for 
optimisation problems such as the image inpainting problem. 


Further possible research: 
e@ Study of the rates of convergence. 


@ Why do the perturbations help? 
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Thank you! 
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